DOCUMENT RESUME 



ED 036 873 



24 



CG 005 172 



AUTHOR 

TITLE 



INSTITUTION 

SPONS AGENCY 

BUREAU NO 
PUB DATE 
CONTRACT 
NOTE 



Postman, Leo 

Comments on Professor Gagne's Paper Entitled 
"Instructional Variables and Learning Outcomes™" 
Center for the Study of Evaluation of Instructional 
Programs Occasional Report No* 18. 

California Univ„, Los Angeles., Center for the Study 
of Evaluation™ 

Office of Education (DHEW) , Washington, D™C™ Bureau 
of Research,, 

BR-6-1646 
Sep 68 

0 EC- 4- 6- 061646-1 909 

15p« ; Paper from the Proceedings of the 
Problems in the Evaluation of Instruction, Los 
Angeles, California, December 13 — 15, 1967 



Symposium on 



EDRS PRICE 
DESCRIPTORS 



EDRS Price MF-$0. 



Goals, *0b ject ives 



5 HC- 


$ 0 „ 8 5 


rn ent , 


^Education 


ophy, 


Evaluation 


ing. 


* Me a sure men 


is in 


agreement 



^Measurement 



AESTRACT 

Professor Postman 
Gagne* s view that valid measurement of learning outcomes is an 
essential part of the evaluation of educational systems. The paper 
stresses that the categorization of outcomes and measuring operations 
should be regarded as flexible heuristic devices, and that it is 
desirable to guard against the risk of standardization. It also 
discusses the classes of outcomes distinguished in experimental 
investigation, and focuses on the selection of operations which will 
both avoid distortion of measurement and distinguish the outcomes 
under study from the action of other variables. (Author) 
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COMMENTS ON PROFESSOR GAGNE’S PAPER ENTITLED 
"INSTRUCTIONAL VARIABLES AND LEARNING OUTCOMES" 

Leo Postman 

Professor Gagne has presented to us a very useful conceptual 
framework for the assessment of learning outcomes. I find myself 
in full agreement with his view that the valid measurement of 
learning outcomes is an essential and, indeed, a critical part of 
the evaluation of educational systems. I want to underscore espe- 
cially his call for the use of measures of retention and of trans- 
fer in the assessment of learning outcomes. Since the objective 
of instruction is preparation for future activities, long-term 
retention and the potentialities for transfer to the mastery of 
new tasks must be the primary criteria of the success of our 
methods of teaching. 

A major value of Professor Gagne’s contribution lies in the 
combination of a general analysis of principles of measurement 
with the development of an orderly system for the classification 
of learning outcomes. There is likely to be wide agreement on 
his formulation of the criteria which must be met by valid meas- 
ures of learning outcomes, namely, distinctiveness and freedom 
from distortion. It is, indeed, essential to differentiate as 
precisely as possible the measuring operations for various classes 
of learning outcomes and to minimize the influence of uncontrolled 
and irrelevant variables on performance. 

We would expect less consensus when we consider the trans- 
lation of these general principles into specific operations. The 
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choice of categories of outcomes and of defining operations is 
based both on theoretical presuppositions and on generalizations 
from avaixable empirical findings. There will be legitimate dif- 
ferences of opinion regarding the most useful categorization of 
learning outcomes and the empirical generalizations supported by 
the experimental evidence. The discussion of such differences in 
opinion should in the long run contribute to the improvement of 
our methods of assessment. With this objective in mind, I should 
now like to comment on some of Professor Gagne’s proposals for 
the application of the basic criteria of measurement to the sys- 
tematic assessment of learning outcomes. My purpose will be not 
so much to disagree with the substance of his recommendations as 
to emphasize the close interrelations between theory, inferences 
from available empirical data, and the establishment of criteria 
for new measurements. The burden of my argument will be that the 
categorization of outcomes and the measuring operations coordinated 
with them should be regarded as flexible heuristic devices, and 
that it will be well to guard against the risk of standardization. 

I believe that is also Professor Gagne’s intention, but it may be 
useful to focus explicitly on this point. 

Let us consider first the measurement of the three broad 
classes of outcomes distinguished traditionally in our experimental 
investigations: learning, retention, and transfer. To measure 

each of these outcomes distinctively requires the performance of 
a control operation to support the conclusion that the observed 
outcome reflects the process under study rather than some other 
process. Once this requirement is accepted, the choice of the 
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appropriate control operations is not necessarily self-evident 
and often must be based on theoretical decisions. Take the distinc- 
tive measurement of transfer as an example. As Professor Gagne" 
formulates the problem, "the inference one wishes to make is that 
some capability a . . . has an effect on the learning of capability 
b. The capabilities a and b are different in some respects that 
are specifiable." To insure distinctiveness, a control group is 
used to demonstrate that without prior acquisition of a there is 
no effect on the learning b. And the control operations must 
be such as to permit the inference that "capability a has trans- 
ferred, not simply that capability b has been facilitated or 
interfered with." 

These defining operations are clearly appropriate to the 
measurement of specific transfer, i.e., transfer based on the 
similarity relations between elements in the successive tasks. 

The capabilities a and b are different in specifiable ways. The 
transfer of a_ influences tVe learning of b because there is some 
systematic relation between these different capabilities. Facil- 
itation of b or interference with b is not included within the 
domain of transfer. The classical paradigms of specific transfer, 
such as the learning of old responses to new stimuli, can be 
readily subjected to tests of distinctiveness implied by these 
boundary conditions. However, it would be difficult to apply these 
particular criteria of distinctive measurement to another type of 
transfer which contributes heavily to the progressive improvement 
of the learner in the performance of successive tasks, viz., 
general transfer or learning to learn. 
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In studies of learning to learn, the experimental procedures 
are explicitly designed to minimize the possibility that the obser- 
ved improvements reflect the transfer of specific components from 
earlier tasks; thus, every effort is made to eliminate overlap of 
elements or similarities between the units in the successive tasks. 
As the individual goes from task A to task B, the corresponding 
capabilities a and b are assumed to be instances of the same 
class with reference to general transfer or learning to learn. 

The question at issue is whether the individual changes his mode 
of attack on B as a result of his experience with A; in that sense 
transfer is the facilitation of b rather than the carry-over of 
a specific a. It is our hope, of course, to be able through 
experimental analysis to break down general transfer into component 
habits and skills. However, in embarking on studies of learn- 
ing to learn we were unable to specify these component habits and 
skills in advance. Rather, the component sources of transfer must 
be induced gradually from the conditions and characteristics of im- 
provement and the validity of the inductions when tested experi- 
mentally. Thus, as our knowledge of general transfer increases, 
the experimental paradigms used in its analysis progressively 
approach those of specific transfer. 

The basic point here is that the control operations designed 
to demonstrate distinctiveness will be different for specific 
transfer and for general transfer. In fact, it is now standard 
procedure in experiments on specific transfer to include control 
operations to parcel out the effects of general transfer. Thus, 

I am in no way questioning the defining operations for transfer 
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proposed by Professor Gagne but am merely trying to delimit the 
domain of outcomes to which they apply. It is important that our 
approach to the problem of transfer be in no way constrained by 
any existing set of criteria of distinctiveness. The same applies, 
of course, to other learning outcomes. It is a truism worth 
emphasizing that theoretical considerations must take precedence 
over the constraints imposed by conventional operational defini- 
tions . 

While the criterion of distinctiveness must be recognized as 
essential for orderly measurement, we must also guard against the 
possibility that a compartmentalization of outcomes unduly restricts 
our inferences about the effects of training. It is possible to 
point to situations in which outcome B provides the best, and per- 
haps the only means for establishing the existence of outcome A 
because the usual criteria for A lack sensitivity. Consider the 
situation in which an individual is given several practice trials 
on a task, say on a list of paired associates. An examination of 
his performance on these practice trials shows that there are a 
number of items which he never gave correctly, and he is also 
unable to recall any of these items correctly on a test of reten- 
tion immediately after the end of practice. By the usual distinc- 
tive criteria, he has failed to learn these items. There is clear 
experimental evidence, however, that under these circumstances 
a test of transfer will show that some degree of learning for these 
items did occur on the practice trials. For example, if the stim- 
uli in these items are paired with new responses, there will be 
significant negative transfer relative to an appropriate control 
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baseline. The degree of learning on the practice trials was insuf- 
ficient for the performance of the prescribed task, but it was great 
enough to result in a significant amount of transfer. It may be 
mentioned in passing that the use of a transfer test for determin- 
ing whether or not learning has occurred has become an accepted 
procedure in experimental evaluations of the hypothesis that asso- 
ciation is an all-or-none process. 

More generally, a transfer test must typically be used to 
decide between alternative interpretations of what an individual 
has learned in a given training situation. The analysis of stim- 
ulus selection is a case in point. Suppose the learning task 
consists of paired associates in which the stimulus terms are non- 
sense syllables presented against colored backgrounds, these 
colors being clearly different for each syllable. Once the sub- 
ject has mastered the list, the question arises of what stimuli 
he has learned to respond to--the syllables, the colors, or both. 
That is, did he select one of the components of the stimulus terms 
and associate his responses to that component? The answer to the 
question of what he has learned depends on whether and to what 
extent there has been such stimulus selection. And the answer 
can be obtained only by a test of transfer, i.e., by determining 
the subject’s ability to give the correct responses to each of 
the components separately and in combination. Thus, the logic 
of the experimental question makes it necessary to draw inferences 
about learning by means of the distinctive operations of transfer. 

It is always necessary to insist that the operations of measure- 
ment be distinctive, but the acceptance of the criterion of 
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distinctiveness does not entail an invariant identification of 
a given type of outcome with a particular set of measuring opera- 
tions . 

When the criterion of distinctiveness is applied to measures 
of outcomes of varying complexity, as is done in the hierarchical 
scheme proposed by Professor Gagne , it is important not to assume 
on a priori grounds that the more complex outcomes represent a 
compounding of the less complex ones . Suppose that a response 
chain has been established and that it is possible to specify 
and to measure independently and distinctively each of the links 
in the chain. We must be cautious about making the inference 
that the integrated chain represents a summation of these inde- 
pendent elements, and the same holds for any capability which can 
be analyzed into distinctive and independently measurable compo- 
nents. The two-stage conception of paired-associate learning to 
which Professor Gagne refers is a case in point. The distinction 
between response learning and associative learning has proved ver 
useful in the analysis of the conditions and characteristics of 
paired-associate learning. Each component can be measured inde- 
pendently, for example, by free recall of the responses on the 
one hand and a test of stimulus -response matching on the other. 
But it is also a fact that at a given stage of practice the sub- 
ject T s performance in the paired-associate task does not repre- 
sent a simple summation of the two components measured independ- 
ently. When a high-order capability is critically dependent on 
the interaction of conceptually distinct components, two-stage 
measurement yields only limited information about the status of 




such a complex capability. The need then arises for the develop- 
ment of distinctive measures directed at the assessment of the 
interaction of the components . 

The decisive influence of theory on the formulation of the 
criteria of valid measurement is exhibited most clearly in rela- 
tion to the requirement of freedom from distortion. To avoid 
distortion of measurement we must choose a set of operations which 
distinguishes the outcome under study from the action of other 
variables. To apply this principle we must then distinguish be- 
tween relevant and irrelevant variables, and that is clearly a 
theoretical decision. Such decisions will obviously be subject to 
disagreement on theoretical grounds. I will limit myself to one 
illustrative example where I found myself in disagreement with 
Professor Gagne’s specification of potential sources of distortion. 
In describing the measuring operations for the assessment of re- 
tention, he states that distortion is prevented by operations which 
control, among other things, the kind of intervening activity known 
to produce varying amounts of interference, i.e., retroactive inhib 
tion. But according to at least one theoretical view, forgetting 
is always the result of interference, whether this interference is 
produced by formal laboratory tasks or by the activities of the 
learner outside the laboratory. On this theoretical assumption it 
becomes impossible in principle to eliminate, or to hold strictly 
constant, interference from intervening activities in the measure- 
ment of long-term retention. One can try to influence the prob- 
ability of interference by choosing materials assumed to have 
certain similarity relations to the learner’s outside activities. 



Clearly, an interference theorist, a decay theorist, and a con- 
solidation theorist will have quite different conceptions of what 
are distortion-free conditions for the measurement of retention. 
Given the lack of theoretical agreement among experimental inves- 
tigators, the same would hold true for most if not all outcomes 
of learning. 

As I indicated at the outset, I find myself in substantial 
agreement with Professor Gagne's basic approach to the measurement 
of learning outcomes. In dissenting from some of his proposals, 
it has been my intention to bring to the fore the inevitable the- 
oretical basis of decisions about measurement. A good theory will 
generate appropriate methods of measurement, but the continued 
acceptance of currently successful measuring operations will not 
necessarily lead to better theories. 
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